Fast multi-spectral image registration by modeling platform motion

ABSTRACT

A method of predicting warps for image registration includes receiving warp data for a first band pair in an N-band system. The method includes fitting a warp model to the warp data to produce an offset profile for the first band pair, predicting a respective offset profile for at least one other band pair in the N-band system, and using the predicted respective offset profile to generate a warp for the at least one other band pair. The method also includes registering images of the at least one other band pair using the warp for the at least one other band pair. In another aspect, a method of predicting warps for image registration includes receiving warp data for m band pairs in an N-band system, wherein m≤(N−1) and m&gt;2, to register images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/482,212 filed Apr. 7, 2017 entitled “FAST MULTI-SPECTRAL IMAGEREGISTRATION BY MODELING PLATFORM MOTION”, which claims the benefit ofU.S. Provisional Patent Application No. 62/321,008 filed Apr. 11, 2016,the contents of which are hereby incorporated by reference in theirentirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under a Contract awardedby an agency. The government has certain rights in the invention.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure relates to imagery, and more particularly toregistration of multiple images such as used in multi-modality andmulti-spectral imagery.

2. Description of Related Art

Registration between the different bands of multi-spectral imagery,e.g., acquired by an airborne or space sensor, is particularlychallenging as the bands that are spectrally far apart do not correlatevery well across the image. For example, bands deep in the IR regionhave both a reflective and an emissive component whereas the bands inthe visual region are purely reflective.

Simple phase correlation to obtain the warp is not very effective inthese instances. Similarly, registering images acquired using differentmodalities such as LiDAR, narrow band spectral, and broad band visualphotographic imagery present the same issues.

Conventional methods and systems have generally been consideredsatisfactory for their intended purpose. However, there is still a needin the art for improved image registration for multi-modal andmulti-spectral images. The present disclosure provides a solution forthis need.

SUMMARY OF THE INVENTION

A method of predicting warps for image registration includes receivingwarp data for a first band pair in an N-band system. The method includesfitting a warp model to the warp data to produce an offset profile forthe first band pair, predicting a respective offset profile for at leastone other band pair in the N-band system, and using the predictedrespective offset profile to generate a warp for the at least one otherband pair. The method also includes registering images of the at leastone other band pair using the warp for the at least one other band pair.

Fitting a warp model to the warp data can include producing a slopeprofile for the first band pair, wherein the method includes predictinga respective slope profile for the at least one other band pair in theN-band system, and wherein using the predicted offset profile togenerate a warp includes using both the predicted offset and slopeprofiles to generate the warp.

Predicting a respective offset profile can include shifting and scalingthe offset profile for the first band pair. Predicting a respectiveoffset profile can include estimating stick spacing for the N-bands of afocal plane array (FPA). Shifting and scaling values of the offsetprofile for the first band pair can be derived from the stick spacing ofthe bands in the first band pair and the bands in the at least one otherband pair. The shifting and scaling values for predicting the at leastone other band pair, d and e, from the first band pair, f and g, can begiven as

${{\Delta \; y_{{de}{({fg})}}} = \frac{y_{d} + y_{e} - y_{f} - y_{g}}{2}},{r_{{de}{({fg})}} = \frac{y_{e} - y_{d}}{y_{g} - y_{f}}},$

respectively, where {y_(d), y_(e), y_(f), y_(g)} denote spatiallocations of bands {d, e, f, g} respectively along a y direction in afocal plane array. Predicting a respective offset profile for the atleast one other band pair, d and e, can include shifting and scaling theoffset profile for the first band pair, f and g.

ĉ _(de(fg))(y)=ĉ _(fg)(y−Δy _(de(fg)))r _(de(fg)),

where ĉ_(fg)(.) denotes offset profile estimated for first band pair fand g. Using the predicted respective offset profile to generate a warpfor the at least one other band pair includes using a respective opticaldistortion profile for each respective band pair.

A method of predicting warps for image registration includes receivingwarp data for m band pairs in an N-band system, wherein m≤(N−1) and m>1,fitting respective warp models to the warp data for each respective oneof the m band pairs to produce respective offset profiles for the m bandpairs, predicting m respective offset profiles for at least one bandpair in the N-band system, combining all the m predicted offset profilesinto a single optimal offset profile for the at least one band pair ofthe N-band system, using the optimal offset profile to generate a warpfor the at least one band pair, and registering images of the at leastone band pair using the warp for the at least one band pair.

Using the predicted respective offset profile to generate a warp for theat least one other band pair can include using a respective opticaldistortion profile for the at least one other band pair. Predicting arespective offset profile can include estimating stick spacing for theN-bands of a focal plane array (FPA). The m offset profiles predictedfor the at least one band pair of the N-band system can include anestimation of uncertainty of each prediction. Combining the mpredictions into an optimal offset profile can take into account theuncertainty of each prediction. The optimal offset profile predicted canbe obtained as a weighted sum of individual m predictions, whereinweights are inversely proportional to respective uncertainty andnormalized to sum to one.

These and other features of the systems and methods of the subjectdisclosure will become more readily apparent to those skilled in the artfrom the following detailed description of the preferred embodimentstaken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

So that those skilled in the art to which the subject disclosureappertains will readily understand how to make and use the devices andmethods of the subject disclosure without undue experimentation,preferred embodiments thereof will be described in detail herein belowwith reference to certain figures, wherein:

FIG. 1 is a schematic view of an exemplary embodiment of a focal planearray (FPA) constructed in accordance with the present disclosure,showing stick spacing for three different sticks or bands of the FPA;

FIGS. 2A-2D are a set of graphs showing spline basis functions inaccordance with an exemplary embodiment of the present disclosure;

FIG. 3 is a data flow diagram of an exemplary embodiment of a method forgenerating calibration parameters in accordance with the presentdisclosure;

FIG. 4 is a data flow diagram of an exemplary embodiment of a method ofpredicting warp for all band pairings using image correlation data froma single band pair in accordance with the present disclosure;

FIG. 5 is a data flow diagram of an exemplary embodiment of a method ofgenerating warp, e.g., optimal warp, using image correlation data fromall band pairs in accordance with the present disclosure; and

FIG. 6 is a data flow diagram of an exemplary embodiment of a method ofgenerating registered bands to a reference image from pairwise warpsgenerated by the exemplary embodiments of FIG. 4 or 5.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made to the drawings wherein like referencenumerals identify similar structural features or aspects of the subjectdisclosure. For purposes of explanation and illustration, and notlimitation, a partial view of an exemplary embodiment of a focal planearray (FPA) in accordance with the disclosure is shown in FIG. 1 and isdesignated generally by reference character 100. Other embodiments ofsystems and methods in accordance with the disclosure, or aspectsthereof, are provided in FIGS. 2A-6, as will be described. The systemsand methods described herein can be used for multi-modal/multi-spectralimage registration.

1 Introduction

Push broom scanner systems acquire imagery by scanning the target areausing platform motion. The focal plane array (FPA) used to acquire theimagery comprises a series of linear sensor arrays (sticks), one foreach spectral band, which are aligned with each other along the lengthof the array but displaced from each other in the orthogonal direction.FIG. 1 shows an example arrangement of the sticks for a 3 band system.The spacing of the sticks in the y direction may be non-uniform and thealignment in the x direction need not be perfect. The push broom systemgenerally positions the FPA for imaging such that the x direction ismore or less perpendicular to the direction of motion of the platformand y aligns with the motion direction. The 2-d imagery of the targetarea is then acquired by reading out the full length of the sensorsticks at regular time intervals (line time) as the platform movesforward in time. The image for each band is put together 1 line at atime until the scanning stops.

Since the sensor sticks are physically displaced from each other alongthe y direction, the images for each band are also displaced from eachother along the y direction. To fully exploit this multi-spectral data,the images from each of the bands needs to be registered to each otherto enable the measurement of the different spectral bands at the samephysical location of the target. If the motion of the platform was heldconstant during the acquisition time, it would be a simple matter offinding a fixed displacement between the band images. In practicehowever, the platform motion cannot be controlled precisely and both thespeed and direction of motion as well as the orientation of the FPA isin a constant state of flux. This variability in motion is more severefor airborne platforms and less variable for spaceborne platforms.

The variable motion introduces a time-varying displacement both in the xand y directions between the different bands and is referred to as awarp between the images. This warp can be estimated by image processingtechniques known to people skilled in the art. One method of estimatingthe warp involves laying down a regular grid of tie-points (spacingdetermined by the expected variability of the motion) in the referenceimage and then finding the corresponding tie-point in the other image bycorrelating the image content in the neighborhood of the tie-point. Sucha process is computationally very intensive and can take a significantamount of processing time for large images and highly variable motion,which requires a fine grid spacing for the tie-points. For a N bandmulti-spectral system, usually one of the bands is deemed as referenceand all the other bands are registered to this reference band. Thisrequires N−1 band pairings that need to be correlated increasing thecomputational burden by a factor of N−1.

Alternatively, the warp may be obtained by employing a model for theimaging system (sensor model) and using it in conjunction with themotion and orientation data of the platform and a measurement of theterrain. However, this method is also very computationally intensive asthe models involved are highly non-linear and do not have closed formsolutions requiring iterative optimization. The accuracy of estimatingthe warp in this manner also suffers from not accounting for the heightof the man-made objects on top of the terrain and the inherent noise inthe measurement of the motion, orientation, and terrain data. Inpractice, the accuracy of the warp obtained solely from a sensor modelis not good enough and people have to revert to image correlationmethods as a final step to obtain the desired accuracy.

Another shortcoming of the conventional image correlation methods isthat all available information is not leveraged to reduce the noise inthe estimated warp. Image correlation treats each pair of bands inisolation when in reality they are all linked together by a commonplatform motion. In a N band system, there are N(N−1)/2 pairs of bandsthat can be made and the warp data from all of these band pairings canbe optimally combined to obtain a superior warp.

The idea we want to leverage is that the warp between any two bandsgives us information about the relative motion of the platform at thetime instances the two bands are acquired. We can use the warp data fromone or more of the band pairings to populate a model of the platformmotion. The relative warp between any other band pairing can then beobtained from the platform motion model. The method can be employed totrade-off computational speed and accuracy of the registration invarying degrees depending on the application requirements. So ifcomputational speed is a concern, we could perform correlation on asingle band pair and use it to infer the platform motion. The warpsbetween all the other bands can then be synthesized from the inferredplatform motion. If registration accuracy is paramount, then warp dataof up to N(N−1)/2 pairs of bands obtained via image correlation may beused to fit the platform motion. The subsequent warp derived from thefitted platform motion is optimal and leverages all availableinformation. In this manner, speed and accuracy can be traded to varyingdegrees by choosing anywhere between 1 and N(N−1)/2 band pairings tocompute the raw warp data.

The motion of the platform occurs in a 6 dimensional space, 3 degrees offreedom for position and 3 degrees of freedom for orientation. However,not all of these degrees of freedom can be estimated from the image thatis constrained to be on the 2-d surface of the terrain. Furthermore,image correlation between the bands only gives us information about therelative motion between the bands and no information about the absolutepositioning of the platform or its orientation. The original motionspace is therefore over parameterized for our purposes. The proposedsolution has two components: first, a compact representation of themotion that most efficiently encodes the warp between any band pairing,and second, a transformation in this compact parameter space that allowsus to compute the warp between any band pairing given the parameters ofthe some other band pairing. The compact representation of the motionfor modeling the warp has another advantage in that it allows us toreduce the noise in the warp estimated by image correlation by utilizinga priori information of the projection geometry.

For the first component of the solution, we propose a separable splinewarp model that encodes the relative motion between any band pairingcompactly in a 4-d space, which represents a reduction of 2 degrees offreedom from the original motion representation. This is covered inSection 2. Section 3 then covers the parameter transformation of thespline warp model that allows us to predict the warp for any bandpairing given the warp parameters for a particular band pairing.

2 Separable Spline Warp Model

In this Section, we formulate a model for the warp between a pair ofbands. One of the bands will be designated as the reference band and theother band will be designated as the warped band. In the interest ofsimplifying notation, we omit the particular band numbers for which thewarp is being computed as the treatment remains the same for any bandpairing.

Let {x_(w), y_(w)} denote the coordinates in the warped band thatcorrespond to the coordinates {x, y} in the reference band. Let

d _(x)(x,y)=x _(w)(x,y)−x  (1)

d _(y)(x,y)=y _(w)(x,y)−y  (2)

denote the space varying displacement (warp) between the warped and thereference band. We assume the images of the reference and warped bandsare acquired by two sensor sticks displaced from each other spatially inthe y direction. The sensor sticks are mounted on a moving platform andthe images are obtained by scanning the sticks in time as the platformmoves. The platform motion may not be perfectly uniform in time and thiscauses a time-dependent warping between the bands. Given the rigidgeometry in the x direction (fast-scan), we expect the warping to belinear in this direction unless there is uncorrected optical distortion.The linear warping in x will be time dependent but the opticaldistortion will be constant in time.

Let s_(x)(.) and c_(x)(.) denote the slope and offset as a function oftime for the linear mapping of the warp in the x component. Similarly,let s_(y)(.) and c_(y)(.) denote the slope and offset for the ycomponent of the warp in time. Let o_(x)(.) and o_(y)(.) denote therelative optical distortion between the warped and the reference bandalong the sensor array for the x and y component respectively. We modelthe space varying translation between the bands as follows

d _(x)(x,y)=s _(x)(y)x+c _(x)(y)+o _(x)(x)

d _(y)(x,y)=s _(y)(y)x+c _(y)(y)+o _(y)(x).  (3)

Note that this is nothing but a Taylor series expansion of the warp d(x,y) along the x direction with the zeroth and first order coefficients asfunctions of y (time) and the higher order coefficients constant intime.

In the subsequent treatment, we will use v={x, y} as an index for x ory. Since the equations are similar for the translations in x and Y, thismakes for compact notation. Assuming the platform motion is more or lesssteady, the slope and offset, s_(v)(.) and c_(v)(.), will be smoothfunctions of time, y. We choose to model them as B-splines and they canwritten as a summation of a set of basis functions as

$\begin{matrix}{{{s_{v}(y)} = {\sum\limits_{j = 1}^{n_{s}}{s_{vj}{B_{j,k}^{s}(y)}}}},} & (4) \\{{{c_{v}(y)} = {\sum\limits_{j = 1}^{n_{c}}{c_{vj}{B_{j,k}^{c}(y)}}}},} & (5)\end{matrix}$

where n_(s) and n_(c) are the number of basis functions that are chosenfor the slope and offset functions respectively. The number of knots andthe spline order k determines the number of basis functions. See Sec.2.1 for generating the set of basis functions B_(j,k)(.), j=1, . . . , ngiven the order and knot locations.

The optical distortion can either be modeled as a B-spline or apolynomial

$\begin{matrix}{{{o_{v}(x)} = {{\sum\limits_{j = 1}^{n_{o}}{o_{vj}{B_{j,k}^{o}(x)}\mspace{14mu} {or}\mspace{14mu} {o_{v}(x)}}} = {\sum\limits_{j = 1}^{n_{o}}{o_{vj}x^{e_{j}}}}}},} & (6)\end{matrix}$

where n_(o) denotes the number of basis functions and e_(j), j=1, . . ., n_(o), denotes the n_(o) exponents chosen for the polynomial basis. Inthe subsequent treatment, we will use B_(j,k) ^(o)(.) to denote eitherfor the spline basis or the polynomial basis.

2.1 Recursive Generation of B-Spline Basis Functions

To generate the B-spline functions, the number and location of the knotsneed to be specified along with the order of the spline. Let the knotssequence be denoted as t₁≤t₂≤ . . . ≤t_(n), where n is the number ofknots. Let k denote the order of the spline where k=1 denotes the 0^(th)order spline. The B-spline basis functions for any order k is generatedrecursively from the previous order basis functions. The knots sequenceis augmented for each order k as follows

$\begin{matrix}{{t^{(k)} = \underset{k\mspace{11mu} {times}}{\underset{}{\{ {t_{l},\ldots \mspace{14mu},t_{1}} }}},t_{2},\ldots \mspace{14mu},t_{n - 1},{\underset{k\; {times}}{\underset{}{ {t_{n},\ldots \mspace{14mu},t_{n}} \}}}.}} & (7)\end{matrix}$

The recursion of the B-spline at any order k>1 is given as

$\begin{matrix}{{{B_{j,k}(x)} = {{\frac{x - t_{j}^{(k)}}{t_{j + k - 1}^{(k)} - t_{j}^{(k)}}{B_{{j - 1},{k - 1}}(x)}} + {\frac{t_{j + k}^{(k)} - x}{t_{j + k}^{(k)} - t_{j + 1}^{(k)}}{B_{j,{k - 1}}(x)}}}},{j = 1},\ldots \mspace{11mu},{n + k - 2},{k > 1},} & (8)\end{matrix}$

and is initialized at k=1 as

$\begin{matrix}{{B_{j,1}(x)} = \{ {\begin{matrix}1 & {{t_{j} \leq x < t_{j + 1}},{j = 1},\ldots \mspace{11mu},{n - 2}} \\1 & {{t_{j} \leq x \leq t_{j + 1}},{j = {n - 1}}} \\0 & {otherwise}\end{matrix},} } & (9)\end{matrix}$

FIGS. 2A-2D show the spline basis functions starting at k=1 and endingat k=4 to obtain the cubic spline. The knots are non-uniformly spaced at[0, 0.25, 0.4, 0.8, 1] and are shown as black dotted lines.

2.2 Warping Model Parameter Estimation

The unknown spline coefficients for slope, offset, and opticaldistortion need to estimated. This is achieved by fitting the warpingmodel Eq. (3) to the warping data obtained by the image correlationalgorithm at a discrete set of tie points. Let {x_(i), y_(i)}, i=1, . .. , m denote a set of m grid points at which the warp has beenestimated. To facilitate the estimation of the unknown coefficient, werewrite Eq. (3) in vector-matrix notation

$\begin{matrix}{\mspace{79mu} {{d = {Ap}},\mspace{20mu} {where}}} & (10) \\{A = \begin{bmatrix}{{B_{1,k}^{s}( y_{1} )}x_{1}} & \ldots & {{B_{n_{s},k}^{s}( y_{1} )}x_{1}} & {B_{1,k}^{c}( y_{1} )} & \ldots & {B_{n_{c},k}^{c}( y_{1} )} & {B_{1,k}^{o}( x_{1} )} & \ldots & {B_{n_{o},k}^{o}( x_{1} )} \\\vdots & \ddots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\{{B_{1,k}^{s}( y_{m} )}x_{m}} & \ldots & {{B_{n_{s},k}^{s}( y_{m} )}x_{m}} & {B_{1,k}^{c}( y_{m} )} & \ldots & {B_{n_{c},k}^{c}( y_{m} )} & {B_{1,k}^{o}( x_{1} )} & \ldots & {B_{n_{o},k}^{o}( x_{,} )}\end{bmatrix}} & (11)\end{matrix}$

is a m×n_(p) matrix, where n_(p)=n_(s)+n_(c)+n_(o), and

$\begin{matrix}{{p = \begin{bmatrix}s_{x\; 1} & s_{y\; 1} \\\vdots & \vdots \\s_{{xn}_{s}} & s_{{yn}_{s}} \\c_{x\; 1} & c_{y\; 1} \\\vdots & \vdots \\c_{{xn}_{c}} & c_{{yn}_{c}} \\o_{x\; 1} & o_{y\; 1} \\\vdots & \vdots \\o_{{xn}_{o}} & o_{{yn}_{o}}\end{bmatrix}},{d = \begin{bmatrix}{d_{x}( {x_{1},y_{1}} )} & {d_{y}( {x_{1},y_{1}} )} \\\vdots & \vdots \\{d_{x}( {x_{m},y_{m}} )} & {d_{y}( {x_{m},y_{m}} )}\end{bmatrix}},} & (12)\end{matrix}$

are n_(p)×2 and m×2 matrices respectively. The matrix P has all thespline coefficients, the matrix d has all the data, and the matrix A isconstructed based on the model structure. The spline basis functionshave a compact local support and this makes the matrix A to be sparse.Note that bold case letters will be used to represent matrices.Subscripts on matrices will be used to identify rows (first index) andcolumns (second index) with * denoting all indices. For example, A_(i*)denotes the i^(th) row and A_(*j) denotes the j^(th) column of matrix A.The first and second columns of p and d matrices contain the modelcoefficients and data for x and y respectively. Hence we will also usep_(*v) or d_(*v) with v taking values {x,y} to denote the column vectorscorresponding to x and Y.

2.2.1 Least Squares Estimation of Spline Coefficients

Equation (10) compactly represents the warp model evaluated at the tiepoint grid where warping data is available either through imageregistration or some other means. In general, there will be noise in thedata either due to measurement process or model mismatch. We would liketo adjust the unknown spline coefficient to minimize the mean squareerror between the model predictions and the data. The least-squaresestimate can be obtained in closed form and is given as

{circumflex over (p)}=(A ^(t) A)⁻¹ A ^(t) d.  (13)

Note that the ̂ notation indicates that the underlying quantity has beenestimated.

2.2.2 Weighted Least Squares Estimation of Spline Coefficients

The solution of Eq. (13) assumes that the noise is uniformly distributedacross all the m data points and that the two components in x and y areindependent. However, depending on how the data values are obtained forthe warping, the noise may be quite variable between the data points andalso coupled between the two components. Let W(i) denote the 2×2weighting matrix for data point i based on the confidence we have on theestimate for that point. The inverse of the noise covariance matrix canbe chosen as the weighting matrix. The weighted mean square error acrossall m data points in vector-matrix notation is given as

$\begin{matrix}{{C = {\sum\limits_{i = 1}^{m}\; {( {{{A(i)}\overset{arrow}{p}} - d_{i^{*}}^{t}} )^{t}{W(i)}( {{{A(i)}\overset{arrow}{p}} - d_{i^{*}}^{t}} )}}},{where}} & (14) \\{{A(i)} = \begin{bmatrix}A_{i^{*}} & 0 \\0 & A_{i^{*}}\end{bmatrix}} & (15)\end{matrix}$

is a 2×2n_(p) matrix and

$\begin{matrix}{\overset{arrow}{p} = \begin{bmatrix}p_{*_{x}} \\p_{*_{y}}\end{bmatrix}} & (16)\end{matrix}$

is a 2n_(p)×1 vector containing the spline coefficient for bothcomponents. The spline coefficients that minimize the mean square errorC is obtained by differentiating Eq. (14) with respect to p and settingit equal to zero

$\begin{matrix}{{{\frac{\partial C}{\partial\overset{arrow}{p}}}_{\overset{arrow}{p} = \hat{\overset{arrow}{p}}} = 0}{{\sum\limits_{i = 1}^{m}\; {2{A^{t}(i)}{W(i)}( {{{A(i)}\hat{\overset{arrow}{p}}} - d_{i^{*}}^{t}} )}} = 0}{{\sum\limits_{i = 1}^{m}\; {{A^{t}(i)}{W(i)}{A(i)}\hat{\overset{arrow}{p}}}} = {\sum\limits_{i = 1}^{m}\; {{A^{t}(i)}{W(i)}d_{i^{*}}^{t}}}}{\hat{\overset{arrow}{p}} = {( {\sum\limits_{i = 1}^{m}\; {{A^{t}(i)}{W(i)}{A(i)}}} )^{- 1}( {\sum\limits_{i = 1}^{m}\; {{A^{t}(i)}{W(i)}d_{i^{*}}^{t}}} )}}} & (17)\end{matrix}$

2.2.3 Special Case for Uncorrelated Components

If the x and y components are uncorrelated (weighting matrices W(i) arediagonal), then Eq. (17) can be simplified since the spline coefficientfor x and y can be estimated independently. Let

$\begin{matrix}{{W_{x} = \begin{bmatrix}W_{11}^{(1)} & \; & 0 \\\; & \ddots & \; \\0 & \; & W_{11}^{(m)}\end{bmatrix}},{W_{y} = \begin{bmatrix}W_{22}^{(1)} & \; & 0 \\\; & \ddots & \; \\0 & \; & {W_{22}^{(m)},}\end{bmatrix}}} & (18)\end{matrix}$

be the m×m diagonal weighting matrices for x and y componentsrespectively. Then the weighted least-squares solution for the twocomponents is given as

{circumflex over (p)} _(*) _(v) =(A ^(t) W _(x) A)⁻¹ A ^(t) W _(v) d_(*) _(v) ,v={x,y}  (19)

Note that until now we have assumed the basis functions are the same forx and y components. This restriction can be easily relaxed by using adifferent matrix A for the x and y components. For the coupled solutiongiven by Eq. (17), the matrix A(i) can be modified as

$\begin{matrix}{{{A(i)} = \begin{bmatrix}A_{{xi}^{*}} & 0 \\0 & A_{y_{i^{*}}}\end{bmatrix}},} & (20)\end{matrix}$

where A_(x) and A_(y) denotes the A matrix formed from the basisfunctions for x and y respectively.

2.2.4 Numerical Computation Considerations

The warping model matrix A can end up being badly scaled andill-conditioned leading to a host of numerical issues in the solution ofthe model spline coefficients. The problem arises because the range of xand y may be large and quite different from each other. The large rangeof x makes the basis functions for the optical distortion model given byEq. (6) to have a very large dynamic range if the polynomial model ischosen. With finite precision, the matrices will become rank deficientmaking it difficult to fit the data correctly. To overcome this problem,it is advisable to first normalize the range of the x and y between 0and 1. The warping model is then fitted in the normalized domain and theresults scaled back to the original domain for subsequent processing.

Also note the solutions in Eqs. (13), (17), and (19), require a matrixinverse. Although they have been written in that form, it is notadvisable to compute the explicit inverse both from a numerical accuracyperspective nor performance considerations. Instead, it is better totake the inverted matrix to the other side of the equation and solve theresulting set of linear equations using Gaussian elimination.

2.3 Warp Model Evaluation

Once the spline coefficients are estimated using eqs. (13) or (17) or(19), the slope, offset, and optical distortion profiles can be computedusing eqs. (4), (5), and (6), as

$\begin{matrix}{{{{\hat{s}}_{v}(y)} = {\sum\limits_{j = 1}^{n_{s}}\; {{\hat{s}}_{vj}{B_{j,k}^{s}(y)}}}},} & (21) \\{{{{\hat{c}}_{v}(y)} = {\sum\limits_{j = 1}^{n_{c}}\; {{\hat{c}}_{vj}{B_{j,k}^{c}(y)}}}},} & (22) \\{{{{\hat{o}}_{v}(x)} = {\sum\limits_{j = 1}^{n_{o}}\; {{\hat{o}}_{vj}{B_{j,k}^{o}(x)}}}},} & (23)\end{matrix}$

The estimated warp can then be evaluated at any spatial location (x, y)in the image as

{circumflex over (d)} _(v)(x,y)=ŝ _(v)(y)x+ĉ _(v)(y)+ô _(v)(x)  (24)

To produce a de-warped band that is registered to the reference band, animage resampling routine will evaluate the warp using eq. (24) at eachpixel location of the reference image to determine the location of thecorresponding pixel in the warped band and then use the interpolatedpixel value of the warped band at this location to fill in the valuesfor the de-warped band at the reference pixel location.

2.4 Uncertainty of the Slope and Offset Profiles

The covariance matrix (uncertainty) of the parameter vector {circumflexover (p)}_(*) _(v) including the slope, offset, and optical distortion,spline coefficients for the case of uncorrelated x and y components canbe computed as

{circumflex over (Σ)}_(v)=(A ^(t) W _(v) A)⁻¹.  (25)

Note that the spline and offset coefficients covariance matrices arecontained as sub-matrices of {circumflex over (Σ)}_(v). The covarianceof the slope spline coefficients is given as

{circumflex over (Σ)}_(sv)={circumflex over (Σ)}_(v1:n) _(s) _(,1:n)_(s) ,  (26)

where we have used the colon notation in the subscript to denote therange of row and column indices to extract from the overall covariance.So for example, the subscript notation in eq. (26) means that the matrixspanning rows indices {1, . . . , n_(s)} and column indices {1, . . . ,n_(s)} within the matrix {circumflex over (Σ)}_(v) is extracted.Similarly, the covariance of the offset spline coefficients is given as

{circumflex over (Σ)}_(cv)={circumflex over (Σ)}_(vn) _(s) _(+1:n) _(s)_(+n) _(c) _(,n) _(s) _(+1:n) _(s) _(+n) _(c) .  (27)

The variance (uncertainty) of the slope profile at a particular timeinstance y is given as

{circumflex over (σ)}_(sv) ²(y)=B _(s)(y){circumflex over (Σ)}_(sv) B_(s) ^(t)(y)  (28)

where

B _(s)(y)=└B _(1,k) ^(s)(y) . . . B _(n) _(s) _(,k) ^(s)(y)┘  (29)

A similar expression applies for the offset profile

{circumflex over (σ)}_(cv) ²(y)=B _(c)(y){circumflex over (Σ)}_(cv) B_(c) ^(t)(y)  (30)

where

B _(c)(y)=└B _(1,k) ^(c)(y) . . . B _(n) _(c) _(,k) ^(c) k(y).┘  (31)

3 Warp Model Parameter Transformation Between Band Pairings

In this Section, we derive a transformation between the slope and offsetprofiles obtained from one pair of bands to another pair of bands. Thiswill allow us to predict the slope and offset profiles for an arbitrarypairing of bands given the slope and offset profiles for a particularband pairing. For notational simplicity, we will omit the subscriptsassociated with the x and y component as the expressions are the samefor both of them. The transformation for the slope and offset profile isalso similar so all the equations below are written only for the slopeprofile. The expressions for the offset profile can be obtained bysimply substituting c for s in the expressions, i.e., this substitutionexplains how to obtain the corresponding expressions of the offsetprofile from the expressions of the slope profile.

Since we are transforming from one band pairing to another, we will needto explicitly specify the bands that are being used in the notation. Soall the quantities derived in previous Sections will have a subscriptdenoting the band pairing used to estimate them. For example, ŝ_(fg)(.)denotes the slope profile estimated for the warp between bands f and gusing image correlation data as given by eq. (21). When we predict theslope profile for a new pair of bands d and e using the slope profileestimated from bands f and g, we will use the subscript notationŝ_(de(fg))(.).

The slope and offset profiles encode the relative motion at the timeinstances the two bands are measured. In this sense, they arederivatives of the platform motion profile. So the slope and offsetprofiles should just scale by the ratio of the time difference betweenthe two band pairings. The profiles will also be shifted in time betweenthe two band pairings. The shift will be equal to the time differencebetween the time spacing of the two band pairings. Taking these twofactors into account, the predicted slope profile of band pairing {d,e}from the slope profile estimated for band pairing {f, g} given as

ŝ _(de(fg))(y)=ŝ _(fg)(y−Δy _(de(fg)))r _(de(fg)),  (32)

where the time shift is

$\begin{matrix}{{{\Delta \; y_{{de}{({fg})}}} = \frac{y_{d} + y_{e} - y_{f} - y_{g}}{2}},} & (33)\end{matrix}$

and the scaling is

$\begin{matrix}{{r_{{de}{({fg})}} = \frac{y_{e} - y_{d}}{y_{g} - y_{f}}},} & (34)\end{matrix}$

and {y_(d), y_(e), y_(f), y_(g)} denote the spatial locations of thebands {d, e, f, g} respectively along the Y direction in the FPA(example shown in FIG. 1). The uncertainty of the predicted slopeprofile is given as

{circumflex over (σ)}_(de(fg)) ²(y)={circumflex over (σ)}_(fg) ²(y−Δy_(de(fg)) ²,  (35)

where {circumflex over (σ)}_(fg) ²(.) denotes the variance of the slopeprofile for band pairing {f,g} estimated from image correlation data asgiven by eq. (28).3.1 Optimal Estimation of Slope and Offset Profile from MultiplePredictions

If image correlation is performed between multiple band pairings toimprove the registration performance, we will have multiple predictionsof the slope and offset profile for each band pairing. These predictionscan be optimally combined given the uncertainty associated with eachprediction to obtain an optimal estimate of the profile, which combinesthe information from all the available data.

There are N(N−1)/2 band pairs in a N band system and we can choose to doimage correlation between any of those combinations. Let M be the numberof band pairs chosen for image correlation. Let P be the set of all bandpairs on which image correlation is performed. Then the optimal estimateof the slope profile for a particular band pair {d,e} is obtained asweighted sum of all the individual M predictions

$\begin{matrix}{{{{\overset{\_}{s}}_{de}(y)} = {( {\sum\limits_{{\{{f,g}\}} \in P}\; {{\hat{\sigma}}_{{de}{({fg})}}^{- 2}(y)}} )^{- 1}{\sum\limits_{{\{{f,g}\}} \in P}\; {{{\hat{\sigma}}_{{de}{({fg})}}^{- 2}(y)}{{\hat{s}}_{{de}{({fg})}}(y)}}}}},} & (36)\end{matrix}$

where the uncertainty associated with each prediction is used as weightand we have used the ⁻ notation to denote the optimal smoothing anddistinguish it from the estimated quantities.

To illustrate this process, we will use the example of a 3 band systemas shown in FIG. 1. In this case, we perform 3 image correlationsbetween band pairs {1,2}, {1,3}, and {2,3}. From the image correlationwarp data, we obtain slope profiles for the 3 band pairs ŝ₁₂(.), ŝ₁₃(.),and ŝ₂₃(.) along with their associated uncertainty as covered in Sec. 2.Now if we consider a particular band pair, e.g. {1,3}, we can obtain 3independent estimates of the slope profiles ŝ₁₃₍₁₂₎(.), ŝ₁₃₍₁₃₎(.), andŝ₁₃₍₂₃₎(.) and their associated uncertainty using eqs. (32) and (35).Note that ŝ₁₃₍₁₃₎(.)=ŝ₁₃(.). The final optimal estimate of the slopeprofile for band pair {1,3} is given as

$\begin{matrix}{{{\overset{\_}{s}}_{13}(y)} = \frac{{{{\hat{\sigma}}_{13}^{- 2}(y)}{{\hat{s}}_{13}(y)}} + {{{\hat{\sigma}}_{13{(23)}}^{- 2}(y)}{{\hat{s}}_{13{(23)}}(y)}} + {{{\hat{\sigma}}_{13{(12)}}^{- 2}(y)}{{\hat{s}}_{13{(12)}}(y)}}}{{{\hat{\sigma}}_{13}^{- 2}(y)} + {{\hat{\sigma}}_{13{(23)}}^{- 2}(y)} + {{\hat{\sigma}}_{13{(12)}}^{- 2}(y)}}} & (37)\end{matrix}$

4 Different Configurations for Warp Estimation

To enable the algorithm to be able to generate a warp for any bandpairing from data for a limited set of band pairs, it first needs to becalibrated. This calibration is performed offline using a set of imagesthat have been acquired for calibration purposes and need be done onlyonce for a particular scanner system. It requires image correlationbetween all sets of band pairs to learn the model parameters that do notvary with time. FIG. 3 shows the data flow during calibration for a 3band system. The 3 sets of band pairs {1,2}, {2,3}, and {1,3}, arecorrelated 12 to generate warp data 13, 14, 15, respectively. This datawill generally be noisy especially in areas where there is little edgeinformation in the image to correlate against. The noisy warp data isthen fitted using the warp model 16 and the slope, offset, and opticaldistortion profiles 18, 30, 32, 24, 26, and 28 are generated. This isdescribed above in Sec. 2. The optical distortion profiles 24, 26, and28 between the band pairs are static and stored for future use. Theslope and offset profiles 18, 30, and 32 should be scaled and shiftedversions of one another as seen in eq. (32). By correlating theseprofiles against one another, the time shift between them can beestimated. The time shift is a function of the stick spacing as seen ineq. (33). Using the 3 time shifts, the 3 stick positions 22 {y₁, y₂, y₃}can be estimated 20. Note that we can only solve for the relativespacing between the sticks so y₁ can be set to zero without loss ofgenerality and the other stick positions {y₂, y₃} can be solved for in aleast squares sense from the 3 time shift measurements.

FIG. 4 shows the on-line data flow for computing the warps between allpairs of a 3 band system using data from just a single imagecorrelation. This configuration would be used if computational speed isvery important and reduced accuracy can be tolerated. Note that theprocess for FIG. 4 is run every time a new image is acquired (on-line)whereas the process for FIG. 3 need run only once (off-line) on apreviously collected image data set to estimate the calibrationparameters subsequently used in the on-line process. In this case, bandpair {1,2} of the acquired image is fed to the image correlation routine120 to generate the noisy warp data 140. The warp model 160 is fitted tothis data to generate the slope, offset, and optical distortion profiles180 and 240. The profiles can then be used in eq. (24) to generate 42the warp 44 at any location for band pair {1,2}. The slope and offsetprofiles 180 from band pair {1,2} are used to predict 34 the slope andoffset profiles 36 for band pairs {2,3} and {1,3} using the calibratedstick spacing 22, shown in FIG. 3. Equation (32) is used to do theprediction. The predicted slope and offset profiles 36 along with thecalibrated optical distortion profiles 38, i.e. calibrated opticaldistortions 26 and 28 from FIG. 3, are then used to compute 40, usingeq. (24), the warps 46 and 48 for band pairs {2,3} and {1,3},respectively, at any location.

FIG. 5 shows the data flow for computing the warps 44, 46, and 48between all pairs of a 3 band system by leveraging all availableinformation. In this case, the image correlation is done between allband pairs and this data is used to optimally predict the parameters ofthe warp model. This configuration would be used if accuracy isimportant and computational speed is of secondary consideration.Similarly to FIG. 4, this is an on-line process that is performed everytime a new image is acquired. The slope, offset, and optical distortionprofiles 180, 300, 320, 240, 260, and 280 are first estimated by fittingthe warp model 16 to the data generated by the image correlation process12 for each band pair ({1,2}, {2,3}, and {1,3}), of the acquired image.The estimation process is the same as shown in FIG. 3 but it isperformed on the newly acquired image. All of the 3 slope and offsetprofiles 180, 300, and 320 are used to predict 340 slope and offsetprofile for each band pair using eq. (32). This provides 3 predictionsfor each band pair, which are optimally combined using eq. (36) toproduce a single smoothed estimate 360 of the slope and offset.

The optical distortion profiles 240, 260, and 280 for each band pair mayalso optionally be smoothed 50 using the calibrated optical distortionprofile 380 (a collection of calibrated optical distortions 24, 26, and28 from FIG. 3) for each respective band pair. In the final step, thesmoothed slope and offset profiles 360 along with the (smoothed) opticaldistortion profile 52 for each band pair is used to evaluate 400 thewarp 440, 460 and 480 for that band pair. FIG. 6 shows the data flow fora registration application that uses the generated warps for the example3 band system. In this case, band 1 is considered as the reference bandand the warped bands 2 and 3 are resampled 54 using the warps betweenband pairs {1,2} and {1,3} respectively. The dewarped bands 2 and 3 arenow spatially registered to the reference band 1.

Although we have described the various applications of the proposedmethod for an example 3 band system, those skilled in the art will nowappreciate how to apply these methods to a N band system. Furthermore,the two applications that have been illustrated are at the two extremesof the computational speed versus accuracy trade-off spectrum. It isstraightforward to trade-off speed and accuracy to a different amountfor a N band system by either increasing or decreasing the number ofimage correlations that are performed.

Those skilled in the art will readily appreciate that a system forregistering images can include a module configured to implement machinereadable instructions to perform one or more of the method embodimentsdescribed above.

As will be appreciated by those skilled in the art, aspects of thepresent embodiments may be embodied as a system, method or computerprogram product. Accordingly, aspects of the present embodiments maytake the form of an entirely hardware embodiment, an entirely softwareembodiment (including firmware, resident software, micro-code, etc.) oran embodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theembodiments. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in a flowchart and/or blockdiagram block or blocks

The methods and systems of the present disclosure, as described aboveand shown in the drawings, provide for imagery with superior propertiesincluding improved multi-band/multi-spectral image registration. Whilethe apparatus and methods of the subject disclosure have been shown anddescribed with reference to preferred embodiments, those skilled in theart will readily appreciate that changes and/or modifications may bemade thereto without departing from the scope of the subject disclosure.

What is claimed is:
 1. A method of predicting warps for imageregistration comprising: receiving warp data for a first band pair in anN-band system; fitting a warp model to the warp data to produce anoffset profile for the first band pair; predicting a respective offsetprofile for at least one other band pair in the N-band system; using thepredicted respective offset profile to generate a warp for the at leastone other band pair; and registering images of the at least one otherband pair using the warp for the at least one other band pair.
 2. Amethod as recited in claim 1, wherein fitting a warp model to the warpdata includes producing a slope profile for the first band pair, whereinthe method includes predicting a respective slope profile for the atleast one other band pair in the N-band system, and wherein using thepredicted offset profile to generate a warp includes using both thepredicted offset and slope profiles to generate the warp.
 3. A method asrecited in claim 1, wherein predicting a respective offset profileincludes shifting and scaling the offset profile for the first bandpair.
 4. A method as recited in claim 3, wherein predicting a respectiveoffset profile includes estimating stick spacing for the N-bands of afocal plane array (FPA).
 5. A method as recited in claim 4, whereinshifting and scaling values of the offset profile for the first bandpair are derived from the stick spacing of the bands in the first bandpair and the bands in the at least one other band pair.
 6. A method asrecited in claim 5, wherein the shifting and scaling values forpredicting the at least one other band pair, d and e, from the firstband pair, f and g, are given as${{\Delta \; y_{{de}{({fg})}}} = \frac{y_{d} + y_{e} - y_{f} - y_{g}}{2}},{r_{{de}{({fg})}} = \frac{y_{e} - y_{d}}{y_{g} - y_{f}}},$respectively, where {y_(d), y_(e), y_(f), y_(g)} denote spatiallocations of bands {d, e, f, g} respectively along a y direction in afocal plane array.
 7. A method as recited in claim 6, wherein predictinga respective offset profile for the at least one other band pair, d ande, includes shifting and scaling the offset profile for the first bandpair, f and gĉ _(de(fg))(y)=ĉ_(fg)(y−Δy _(de(fg)))r _(de(fg)), where ĉ_(fg)(.)denotes offset profile estimated for first band pair f and g.
 8. Amethod as recited in claim 1, wherein using the predicted respectiveoffset profile to generate a warp for the at least one other band pairincludes using a respective optical distortion profile for eachrespective band pair.
 9. A method of predicting warps for imageregistration comprising: receiving warp data for m band pairs in anN-band system, wherein m≤(N−1) and m>1; fitting respective warp modelsto the warp data for each respective one of the m band pairs to producerespective offset profiles for the m band pairs; predicting m respectiveoffset profiles for at least one band pair in the N-band system;combining all the m predicted offset profiles into a single optimaloffset profile for the at least one band pair of the N-band system;using the optimal offset profile to generate a warp for the at least oneband pair, and registering images of the at least one band pair usingthe warp for the at least one band pair.
 10. A method as recited inclaim 9, wherein using the predicted respective offset profile togenerate a warp for the at least one other band pair includes using arespective optical distortion profile for the at least one other bandpair.
 11. A method as recited in claim 9, wherein predicting arespective offset profile includes estimating stick spacing for theN-bands of a focal plane array (FPA).
 12. A method as recited in claim11, wherein the m offset profiles predicted for the at least one bandpair of the N-band system include an estimation of uncertainty of eachprediction.
 13. A method as recited in claim 12, wherein combining the mpredictions into an optimal offset profile takes into account theuncertainty of each prediction.
 14. A method as recited in claim 13,wherein the optimal offset profile predicted is obtained as a weightedsum of individual m predictions, wherein weights are inverselyproportional to respective uncertainty and normalized to sum to one.